Optimal Resource Allocation of Cloud-Based Spark Applications

Nowadays, the big data paradigm is consolidating its central position in industry, as well society at large. Lots of applications, across disparate domains, operate on huge amounts and offer great advantages both for business research. According to analysts, cloud computing adoption steadily increasing support analyses Spark expected take a prominent market next decade. As applications gain more importance over time given dynamic nature resources, it fundamental develop an intelligent resource management system provide Quality Service guarantees end-users. This article presents set run-time optimization-based policies advanced analytics. Users submit characterized by priority hard or soft deadline. Optimization address two scenarios: i) identification minimum capacity run application within deadline; ii) re-balance resources case heavy load, minimising weighted deadline tardiness. The solution relies initial non-linear programming model formulation search space exploration based simulation-optimization procedures. execution times are estimated relying gamut techniques, including machine learning, approximated analyses, simulation. benefits approach evaluated Microsoft Azure HDInsight private cluster POWER8 considering TPC-DS industry benchmark SparkBench. results obtained first scenario demonstrate that percentage error prediction optimal usage with respect measurement exhaustive range 4–29 percent while literature-based techniques present average 6–63 percent. Moreover, second scenario, proposed algorithms can complex problems like redistribution among tens less than minute 8 average. On same considered tests, approaches obtain about 57